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Evaluating Web-Based Learning and Instruction (WBLI): 
A Case Study and Framework for Evaluation 

Greg V. Michalski, Ph.D. 



Introduction 

The purpose of this paper is to suggest an alternative 
approach to perform relevant and useful evaluations of 
web-based learning and instruction (WBLI) that will 
accommodate performance and keep pace with the 
growing capabilities of the internet. Rather than focus on 
the particulars of the many specific techniques and 
approaches available to simply collect user-supplied data 
(e.g., using web-based surveys), however, the following 
discussion is concerned primarily with defining (at least 
the precursors of) an evaluative framework that can 
accommodate a range of WBLI interventions specifically 
within a business environment by more deeply exploring 
learner needs, concerns, and expectations. A case study 
involving WBLI designed to assist the E-Service 
solutions group at Nortel Networks implement Time to 
Market (TTM) development principles is used to illustrate 
several aspects of the emergent framework. 

Web-Based Learning and Instruction 

The evolution of the internet and in particular the World 
Wide Web (WWW) continues to expand opportunities to 
deploy WBLI on an unprecedented scale. For example, 
leading market research firms estimate the optical internet 
market will grow to $35 billion (US) by 2001. 

Industry research shows that WBLI involving multimedia 
is a cost-effective communication method — a key to 
travel and traditional training savings that is being used 
by an increasing number of firms and organizations. For 
example, in the 12 months following the deployment of 
its unicast service in 1997, Boeing has enjoyed cost 
savings on training, productivity, travel avoidance, and 
tape distribution citing savings from duplicating and 
distributing videotapes alone totaling $140K/month. 
Their Unicast service handles 400-500 short video clips 
on training, announcements, procedures, and so on. It has 
been receiving 3000-6000 visits per week, serving 20,000 
users. Other companies have made similar claims. MCI 
claimed to achieve $1M in travel reductions within 9 
months, while GE information services claims benefits 
based on time-value of information for marketing, as well 
as savings to train newly hired employees. 

Some commonly cited advantages of WBLI include 

• easy version and integrity control of information 
with global updating capabilities 

• provides a highly-accessible means to capture 
valuable organizational knowledge 



• leverages the use of internal organizational 
networks already in place and familiar to users 

Multimedia and Streaming Use in WBLI 

The most advanced WBLI applications increasingly 
incorporate the use of ’’streamed' 1 video, audio, illustrated 
graphics, and audio synchronous presentation services 
onto desktop PCs. Streaming allows users to view video, 
audio, or enhanced presentation charts while the file is 
being downloaded. 

Two primary types of streaming include unicast and 
multicast applications. Unicast is a one-to-one delivery 
method by which each user is served individually (i.e., the 
server streams requested files to each user so that 100 
requests result in 100 streams). Applications might 
include shared learning or virtual classroom 
environments. Many organizations provide primarily 
unicast service with supplemental "cache servers" in key 
locations so users can stream their content from the 
nearest server to enable the highest quality service 
possible. 

Multicast is a one-to-many delivery method in which a 
single server streams one file to multiple requestors. This 
streamed file can then be replicated and redistributed by 
local servers to save bandwidth. Applications might 
include lectures, presentations, meetings, news headlines 
and weather updates. Both unicast and multicast 
streaming techniques rely upon steadily increasing speed 
and bandwidth which is at the heart of internet growth. 

Building the Foundation of WBLI Capability 

According to Roth (1998) the doubling capability of the 
internet every 12 months seems to be following Metcalfs 
Law. This is named for Robert Metcalf (also known as 
the father of Ethernet) founder of 3Com Corporation. 
According to Roth, "Metcalfs Law states that when you 
connect any number, n, of machines, you get n squared 
potential value." (p. 293) By providing the infrastructure 
for computers to communicate with each other 
(particularly via the growing internet) their value grows at 
an exponential rate. 

Currently there are a handful of companies that constitute 
the main players building the actual roadbed or 
’’infostructure" of the so-called information 
superhighway. Among these, Nortel Networks, has built 
32 out of 40 national and pan-European optical networks 
announced over the past two years globally. These high- 
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performance optical internet solutions, include Nortel's 
OPTera Metro, OPTera LH and the industry-leading 10 
Gbps (giga — or billion — bits per second) platform which 
currently supports as much as 320 billion bits of data per 
second on a single optical fiber. Future capabilities will 
allow this system to deliver 6.4 tera (trillion) bits per 
second on a single fiber. This is notable because in the 
mid 1990s all the phone networks in the world combined 
carried only an average of 1 tera bit per second. Two tera 
bits is equivalent to all the voice traffic in North America 
switched and routed each second. This transfer rate 
would allow the entire contents of the U.S. Library of 
Congress to be transmitted across the country in 
approximately 20 seconds. 

Optical business is growing rapidly as service providers 
(e.g., telecommunications access providers) race to meet 
explosive demands for bandwidth driven by the internet 
and growth in electronic business (e-business). These 
networks are the foundation for the high-performance 
internet that will continue to deliver increased speed, 
reliability, quality and new economics for information 
exchange and distance learning applications such as 
WBLI. 

Against this dynamic backdrop of growing capability the 
design, deployment, and evaluation of WBLI is also 
changing. As outdated approaches using the internet 
mainly to deliver largely passive online slide shows 
become replaced by the more sophisticated multimedia 
presentations that include the streamed applications 
mentioned as well as real-time interactive audio and video 
communications capabilities, the capability and capacity 
to evaluate WBLI will also grow. 

WBLI Evaluation Approaches: A Brief Review 

Several methods and approaches to evaluating web-based 
training have been described in the literature. For 

example, Hall (1997) discussed ten main criteria used for 
semiannual judging of the Multimedia and Internet 
Training Awards (MITA). These include (1) content, (2) 
instructional design, (3) interactivity, (4) navigation, (5) 
motivational components, (6) use of media, (7) 
evaluation, (8) aesthetics, (9) record keeping, and (10) 
tone. 

Within these ten criteria content refers to the 
appropriateness, amount, and quality of the basic 
information included in the training. Instructional design 
focuses on the assembly of the information in such a way 
so as to promote participant learning. Navigation refers to 
the steps taken and the accommodations made to allow 
users to effectively guide themselves through the content 
of the instruction. This involves considerations of the 
presence of a course map as well as appropriate use of 
labels and icons for navigation of the site. Motivational 



components include elements to engage users through the 
use of novelty, humor, adventure, surprises and so on. 
These are the traditional WIIFM (what's in it for me} 
elements included in all well-designed instruction. Use of 
media involves the effective and appropriate use of 
graphics, animation, and multimedia in general. 
Aesthetics refers to what might be alternately termed the 
"face validity" of the instructional site. This involves 
judgments about how credible, attractive and visually 
appealing the instruction is to the visual and auditory 
senses. Record keeping involves data management 
considerations about how any user/leamer performance 
data are routed and ultimately used. Tone refers to the 
degree to which the learning program is appropriate for its 
intended audience. 

The MITA criterion labeled evaluation focuses on 
user/leaner performance on some predetermined exercise 
intended to examine learning or mastery of a topic or 
series of related topics. According to Hall (1997) 
evaluation evidence . might include user/leamer 
completion of a simulation or quizzes, tests, and 
examinations intended to probe learner knowledge related 
to the training. As will become apparent shortly, many 
WBLI and distance learning evaluation efforts focus 
almost exclusively on knowledge testing. 

While these ten criteria emphasize primarily the technical 
aspects of the instructional web site, others have taken a 
somewhat broader view by including considerations of 
both the technical and human infrastructure associated 
with web-based instruction. In addition to instructional 
and web-site design, McGreal (1997) includes 
considerations of roles and responsibilities of several 
participant groups including instructors, tutors, as well as 
learners. According to McGreal (p. 70) "You can never 
make the respective roles and responsibilities of the 
teacher tutors, and students too clear." In connection with 
defining such roles, questions related to control of the 
development and deployment of web-based instruction 
were posed. These ask respectively about the control of 
course development (e.g., individual, cooperative, open, 
teams, contracted out, specialists) and the control of the 
deployment (e.g., self-paced, tutorial, deadlines, 
scheduled labs, live class sessions). 

Other authors have emphasized several dimensions of 
learner evaluation related to WBLI. In describing 
networked learning environments, Chute, Sayers, and 
Gardner (1997) stated that a course evaluation system 
should be designed to provide testing results quickly to 
learners. In addition these authors emphasized the 
important role of support services designed to facilitate 
and assist learners in successfully completing a given 
instructional module. Beyond traditional individualized 
learner tests and quizzes, however, these authors also 
pointed out that collaboration on quizzes "can be a very 
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positive learning activity.” (p. 79) This is because when 
several learners collaborate they are interacting in a 
manner that would not have taken place using a 
traditional (individualized) quiz scenario. Through such 
collaboration learners are exposed to other viewpoints of 
a subject, thus enhancing group-level learning. 

Several authors have discussed the evaluation of distance 
education as a more general enterprise that includes a 
range of delivery technologies in addition to WBLI. In 
defining and applying program evaluation principles to 
the evaluation of distance education Simonson (1997) 
differentiated between theory-based research and 
evaluation noting that evaluation (contrasted to research) 
is the "systematic investigation of the worth or merit of an 
object." (p. 88) In further drawing upon the (1994) Joint 
Committee on Standards for Educational Evaluation, 
Simonson noted that program evaluation is the systematic 
investigation of the worth of an ongoing or continuing 
distance education activity. This author also reviewed 
two approaches to distance education evaluation based 
upon principles of program evaluation. 

Referencing prior work by Woodley and Kirkwood 
(1986), Simonson outlined six categories of evaluation 
information that can be collected about distance education 
programs. The first category includes measures of 
activity which include counts of events, people, and 
objects (often available from administrative records). 
Typical counts might include the number of courses 
produced, the number of students enrolled, or the number 
of applicants for a particular program. The second 
category contains measures of efficiency. Such measures 
could include data pertaining to the number of students 
successfully completing the course, average student 
workloads, the number of students enrolled in related 
courses, and course costs and revenues (as tuition 
generated). The third category describes measures of 
outcomes related to student learning. In addition to test 
scores, student interviews and surveys can be used as well 
as more indirect measures, for example, the 
documentation of the use of courses and course materials 
by other institutions. The fourth category includes 
measures of program aims which might examine data 
about the scope and demographic characteristics of 
learners reached by the program. The fifth category 
involves measures of policy. Policy evaluation often 
resembles market research focusing on surveys of 
prospective students and employers. But policy 
evaluation can also be used to determine the success of 
experimental programs as well. The sixth category 
includes measures of organizations. This involves 
monitoring program effects related to organizational 
efficiency. This can be accomplished through site visits 
and selective interviews people in the organization. 



Another approach useful to evaluate distance learning 
programs is based on an examination of five key areas 
that include accountability , effectiveness , impact , 

organizational context , and unanticipated consequences 
(AEIOU). Within this approach accountability examines 
whether the project planners did what they promised. 
Measures of this might include monitoring the number of 
students who enrolled in or otherwise engaged a program. 
Interviews of program or project staff as well as project 
administrative records provide a good source of 
accountability information. Effectiveness refers to the 
quality of the overall project especially in terms of 
participant attitudes and knowledge. Grading, testing, and 
attitudinal inventories can all provide measures of 
effectiveness. These might include standardized 
measures of achievement as well as participant 
perceptions of the program. Impact measures whether the 
project made a difference. Here the focus in on any 
changes that can be linked directly to the project. 
Methods appropriate to monitoring impact can be either 
qualitative (interviews, focus groups, direct observations) 
or quantitative (surveys, standardized test scores, 
participant record data). Organizational context examines 
which structures, policies, or events in the organization or 
environment helped or hindered the project in 
accomplishing its intended goals. Here attention is paid 
particularly to which factors helped or hindered the 
implementation of a project. Unanticipated consequences 
refer to any unexpected or unplanned changes (either 
positive or negative) that occurred as a result of the 
project. These may be described in terms of anecdotal 
information emerging from interview, focus group, 
journal, or survey data collected after the learning 
program (ex post facto) or WBLI event. The following 
case example includes the use of both quantitative 
(survey) and qualitative (focus group) techniques to 
assess learner perceptions and expectations of a web- 
based instructional course. 

A Case Example 

This case example describes the evaluation of a WBLI 
intervention designed to promote TTM processes within 
an E-service group at Nortel Networks. The E-service 
group (at the time this article was written) consisted of 27 
regular, full-time employees. The vision of this group is 
to provide unified processes and systems enabling web- 
based customer network/business service and support. 
The group's supporting mission is to give customers and 
service personnel consistent, simplified, web-based 
processes and systems that reduce costs, drive revenues, 
and enhance customer satisfaction. The goals of the E- 
service group are to decrease the cost of service, reduce 
the cost of product ownership, and improve customer 
problem resolution through self-service (web-based) 
mechanisms designed to increase the speed of business 
transactions. E-service also aims to facilitate knowledge 
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transfer to both internal and external clients and 
customers, drive revenues by "upselling" services and 
linking to product sales, and increase customer loyalty 
and retention. Common E-service elements include case 
and knowledge management using a common 
infrastructure to mitigate vendor risk (e.g., key word 
search, FAQ, status tracking), documentation access, and 
"searchability" providing a single interface to customers 
with seamless access. Key services and capabilities 
include asset tracking, repair and return as an on-line 
process, software delivery (tied in with order 

management), incremental (patch/fix) updates, beta 
software, real-time updates, and other information 
delivery such as hardware baseline files, bulletins, bar 
code lists, and metrics (e.g., customer satisfaction, usage 
statistics, service revenues, entitlement information, chat 
rooms, and product access/simulation. Although the E- 
service group from its inception has operated in a 
managed project environment, its early recognition of the 
benefits of adopting a formal TTM approach led it to seek 
formal instruction in this area. Before describing E- 
service's experience, however, the following section 
provides a brief overview of TTM and its key principles. 

Time to Market (TTM) 

With the onset of the internet age and the dawn of e- 
business the speed in which products are developed and 
delivered to market has become critical. Accelerating 
delivery of the right product to market is a requirement to 
compete in the data world of short cycle time and is 
critical to achieving success and enhanced 
competitiveness. 

Beginning in the mid- 1990's several bench marking 
studies indicated that it took companies such as Nortel 
more that double the time to introduce new products to 
market than so-called best in class companies. However, 
additional studies and empirical evidence also indicated 
that time to market improvements of 40% to 60% were 
achievable by introducing a disciplined product 
development processes (TTM) into business units (see 
additionally, e.g., Gates, 1999; Harreld, 1994; McGrath, 
1995; Meyer, 1993). The overall TTM project objective 
was to cut Nortel’s new product development cycle time 
in half. This project was aimed at all business units 
seeking dramatic reductions in time to market for both 
products and services. Such reductions would offer 
significant earnings potential driven through research and 
development (R&D) efficiency, reductions in wasted 
R&D spending, increased revenues (first/early to market) 
and improved product margins. 

TTM was fully deployed to all business units throughout 
1999, with the expectation that all business units would be 
self-sustaining (no longer require training and facilitation 
from the core team) by the end of 1999. To date nine 



major business units have engaged TTM processes with 
all business units scheduled to initiate the formation of 
project teams. Metrics were implemented to track project 
progress and its impact. During the year 2000, a small 
TTM core team of best in class process experts will 
facilitate an executive TTM council and a TTM 
practitioner council to ensure that ongoing cross business 
unit collaborative meetings happen, and that a common 
TTM framework is maintained and continuously 
improved. 

TTM BUSINESS CASE: Early estimates indicated that 
the reduction of new product development cycle time by 
50% would generate significant benefits in several areas 
including financial as well as shareholder, customer, and 
employee value. Improved financial results would come 
from increased profitability through improved R&D 
efficiency and productivity reinvestment to decrease 
development cost and increase revenues. Improved 
product margins would also result from being early to 
market by achieving price premiums and getting a jump 
on cost reductions as well as the reduction of wasted 
R&D spending through managed project cancellations 
based on market and customer needs. Shareholder Value 
improvements realized through TTM will lead to a more 
responsive, customer focused company with improved 
earnings potential. TTM impacts customer value by 
facilitating delivery of the right combination of products 
and features. Higher quality and timely delivery are 
driven by more extensive customer involvement which 
fosters greater customer loyalty. TTM impacts employee 
value by promoting empowered integrated project teams 
operating with efficient resource utilization and 
reassignment as well as integrated design and rationalized 
toolsets to generate better skill attraction and retention 
and, ultimately, greater employee satisfaction. 

The TTM process also contains a number of key 
assumptions. There involve the following five areas 

• Cycle Time — Cycle time reduction of 50% (from 
18 month average product development cycle 
time to 9 month average) 

• Earlier Market Delivery — 9 months early 
revenue for new product introductions; 10% 
additional revenue for product life on new 
product introductions; 5% increase in product 
margins for new product introductions 

• Development Productivity — 50% reduction in 
cycle time equating to a 20% productivity 
improvement in new product development 

• Wasted Development Spending — 5% 
development cost savings in ‘99; 6% in 2000; 
7% in 2001 

• Timing — benefits are built on a phased 

deployment plan to engage all business units by 
end of 1999; 50% of business unit development 
projects impacted in first cycle; development 
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projects receive 75% of benefits in first cycle; 
All projects impacted in second cycle with 90% 
of benefits 

TTM PROCESS MODEL: TTM consists of five basic 
elements, each depending on the other for maximum 
impact. These include the establishment of several teams 
including a Portfolio Management Team (PMT) and core 
teams. Other process elements include the definition of 
Business Decision Points (BDP), a Structured 
Development Process, and Integrated Enablers. These 
elements are depicted and briefly described next as 
illustrated in Figure 1. 



The structure of each core team is tailored to the needs of 
the specific project. The extended core team is a broad 
membership of representatives from all functional 
organizations that serve the core team for the particular 
project. Members of the extended core team receive daily 
project direction from the core team member representing 
their function, and, receive technical direction and daily 
administrative management from their functional 
manager. Business Decision Points (BDP) are the 
mechanism by which the PMT reviews, approves, and 
allocates resources to product development projects. 



Portfolio Management Team (PMT) 

• General Manager-led 

• Market timing priority 

• Clear, compelling product roadmap 

• Resource balance across pipeline 



Business Decision 
Point Process 

• Event -driven 

• Business perspective 

• Phase-by-phase funding 

• Project team empowerment 




Structured 

Development Process 

• Common basic workflow 

• Multiple, flexible paths 

• Lead customer engagement 

• Inter-functional linkages 



Integrated Project Teams 

• Small, cross-functional 

• Empowered and accountable 

• Early customer involvement 

• Team performance assessment 



Figure 1. TTM Process Model. 

The Portfolio Management Team (PMT) is the team of 
cabinet-level managers responsible for overseeing the 
pipeline of new product development efforts and 
committing the resources necessary for effective 
execution. The PMT is guided in its decisions by strategic 
and financial goals and must constantly balance the needs 
of multiple development projects at various phases in the 
development process. Through business decision points, 
the PMT empowers core teams to develop and deliver 
new products. 

Integrated Project Teams (IPT) are cross-functional teams 
with full responsibility and authority for executing 
projects. Each IPT is further divided into the core team 
and the extended core team. The core team drives 
decisions and activities from concept approval through 
stable production, and its limited membership (4-8 
people) represents and drives all functional resources 
involved throughout the phases of product development. 
Core teams are approved by and accountable to the PMT. 



BDPs are conducted from a business perspective and 
provide the PMT with the appropriate information to 
make decisions regarding the continuance, redirection, or 
cancellation of projects. 

The Structured Development Process maps the path that 
all projects take in progressing from initial concept 
through general availability. For explanatory purposes, 
this path is organized in a hierarchy of phases, steps, 
activities, and tasks. The intent of the structured 
development process is to help core teams successfully 
plan and manage product development projects. 
Additionally, the process guides core teams in completing 
each successive phase and presenting salient business 
issues to the PMT. 

Integrated Enablers provide core teams with tools and 
functional techniques to engage customer input, plan 
projects, and manage risks. For example. Customer 
Focused Design (CFD) is a methodology integrated into 
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TTM. Its purpose is to provide tools that lead to greater 
understanding of key customer requirements, which are 
then incorporated in the solution design. Other important 
enablers include metrics management, decision-making 
tools, human resources processes, budgeting processes, 
and so on. 

In summary, TTM works because all product programs 
are managed as end-to-end projects with each project 
having an end-to-end accountable leader. Cross- 
functional teams have the project's business success as 
their only objective, and individuals identify with the 
project not with their function. TTM also ensures that all 
project plans are customer focused. Each project has a 
charter defining priorities, desired outcomes, and 
timeframes. The product delivery process uses Alpha- 
Beta terminology and each project defines Alpha-Beta in 
its own context. The general managers and their cabinets 
are responsible for project starts, stops, and release to 
market based on business needs. TTM also involves 
strong executive-level support in which general managers 
are completely on board and personally driving change 
based on customer, market, and business requirements. 

Evaluating Web-Based TTM Instruction for E-Service 

The principles and processes associated with TTM were 
captured in a self-paced, interactive, WBLI module that 
also incorporated multimedia (audio/video) components. 
This module was prepared as a general introduction to 
TTM and was also used by other corporate groups and 
individuals as a means to become quickly oriented to the 
basics of TTM. 

Rather than simply direct E-Service employees to the web 
site containing the TTM module, however, employees 
were instead directed to complete a pre-engagement 
(web-based) questionnaire designed to capture 
demographic, background, and expectations information 
about E-Service employees engaging TTM instruction. 
The questionnaire contained three main sections. The 
first of these contained a series of questions inquiring 
about employee background and interest in TTM as well 
as an item to probe learner expectations of the TTM 
web-based instructional module. Sample items from this 
section include the following: 

I am engaging this Time to Market (TTM) presentation 
because ... (choose one or more): 

□ I want to improve my general knowledge about TTM. 

□ I want to avoid having to attend a traditional, 
classroom- based, course on TTM. 

□ I am just curious about TTM. 

□ I am just curious about the website. 

□ I was encouraged to do so by a co-worker (peer) 

□ I was encouraged to do so by my manager 

□ (of) another reason (please describe below) 



- 6 - 

After completing this presentation I expect to be able to... 
(choose one or more): 

□ Describe basic TTM concept/principles 

□ Answer basic questions about TTM 

□ Understand how TTM supports Nortel business goals 

□ Understand how to apply TTM knowledge to my job 

□ Apply what I have learned on the job 

□ other (please describe below) 

My level of knowledge about Time to Market (TTM) is... 
(choose one): 

□ This is the first time I have heard of TTM. 

□ I have heard of TTM but know extremely little about 
it. 

□ 1 have heard of TTM and I know some its basic 
concepts/principles. 

□ I have a working knowledge of TTM. 

□ I have an extremely high degree of knowledge about 
TTM. 

My level of interest in learning more about TTM is... 
(choose one): 

□ low 

□ moderate 

□ high 

Section two of the questionnaire inquired about learner 
preferences regarding media. Each respondent was asked 
to rate a list of learning methods/media in terms of how 
strongly he or she agreed (using a 5-point Likert scale) 
with it as a preferred means to learn. This list included 
the following 12 items: 

traditional classroom 
audio tape 
video tape 

CD-ROM (multimedia presentation) 
web/internet based (multimedia presentation) 
electronic performance support system (EPSS) 
traditional printed media (books, manuals, etc.) 
radio broadcast 
television broadcast 
satellite based (interactive multimedia) 
desktop (interactive) videoconferencing 
audio conferencing by phone 

The third main section of the pre-engagement 
questionnaire inquired about learner background and 
demographic characteristics such as job role (manager, 
non-manager, hierarchical level within specialization, and 
so on), line of business, age category (in 5 year 
increments from 20 years through over 55 years of age), 
and highest academic degree achieved (non-degree, 
bachelors, masters, doctorate, and other specialized 
degree). 
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This information was collected as a pre-assessment of 
specific learner background characteristics, expectations, 
and experiences regarding TTM especially with the 
context of their work group. While a detailed analysis 
and discussion of the results obtained for the E-Service is 
beyond the scope of the current paper, the reader is 
directed to Michalski (2000) for such a discussion 
involving a larger group sample using the survey items 
described above. The following section discusses the 
results of two focus groups involving both E-Service 
management and non- management employees within the 
E-Service group. 

Focus Group Results 

Although results from the pre-engagement instrument 
were analyzed and reported to E-Service management 
regarding the expectations and readiness of their 
employees to engage TTM WBLI, two separate focus 
groups (facilitated by the author) were performed to more 
completely investigate and understand these results. Each 
session was implemented as a half-day (4 hour) meeting 
among E-Service personnel. 

The first of these sessions involved E-Service 
management and consisted of a total of 9 individuals. 
Having reviewed the TTM web-based instruction, this 
group was primarily interested in the business benefits 
promised by TTM. Discussion centered around topics 
such as which other groups had adopted TTM and what 
has been their experience with the process, as well as the 
mechanics of TTM implementation specifically within the 
context of their highly specialized group. Typical 
questions raised and discussed included 

• What other groups have already adopted TTM? 

• What has been the experience of these groups? 

• How has TTM assisted these groups in more 
effectively accomplishing their goals and 
business objectives? 

• What level of resources and expenditures were 
required to effectively implement TTM? 

• What type of instructional and documentation 
support was available during the implementation 
of TTM? 

Judging from focus group discussions, this group of 
managers was obviously ’’sold” on the idea of TTM and 
sought largely to understand how to best manage the 
implementation of TTM for the employees in their group. 

The second focus session involved a larger number of 
predominantly non-management E-Service employees. 
These were mostly people with highly specialized 
technical expertise related to, for example, web-based 
design and data management, product sustaining and 



support, customer service, marketing, and various other 
non-managerial roles. 

In contrast to the managers participating in the first focus 
session, this group was much less ’’sold” on the idea of 
TTM. Even having engaged the TTM web-based 
instruction (as had the managers), individuals in the non- 
management group had far more questions related to the 
necessity of adopting and, hence, needing to learn about 
TTM in the first place. Typical questions discussed by 
this group included 

• How does TTM differ from basic project 
management? 

• What would need to be actually done differently 
by employees to effectively practice TTM? 

• Has the adoption of TTM been ” mandated ” by 
upper management? 

• What might be the consequences of not adopting 
TTM? 

• What types of metrics were available to monitor 
their group's progress on adopting TTM? 

Overall, this group was much more reserved and reluctant 
to open up and share their views. This situation required 
substantially more effort on the part of the facilitator to 
try to develop the group's level of discussion about TTM. 
Toward the end of the session (in the last 45 minutes or 
so) the group finally did open up more posing the 
following specific comments/questions/concems: 

• What are the specific roles and responsibilities 
of employee relative to TTM? 

• What TTM tools were available? 

• Was there a way to "quick-start” the TTM 
process in the group that would avert the need 
for specialized training on the subject? 

• What were the ” next steps” involved in adopting 
TTM? 

• What were the implications of adopting TTM in a 
program-based (E-Service ) versus project-based 
business environment? 

By the conclusion of both focus group sessions it was 
obvious that there were several areas of employee concern 
that were not previously anticipated. Knowing this, the 
performance of a simple ex post facto evaluation of the 
TTM WBLI (e.g., using knowledge testing and 
satisfaction measures) would have clearly yielded 
misleading results especially based on the employee 
skepticism described above. Using these results the 
following section proposes an approach to evaluating 
WBLI that specifically considers stakeholder group 
differences and uses data describing these as an input for 
effective decision making regarding the merit and worth 
of web-based instruction within an emergent framework. 
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Toward an Evaluation Framework for WBLI 

The case example presented makes it clear that WBLI 
evaluation involves a hybrid of organizational, human, 
and technological factors which all combine to influence 
learning and ultimately performance. Yet most current 
models of distance learning evaluation in general, and 
web-based learning in particular, tend to ignore, diminish, 
or overlook the complex interrelationships between these 
factors. In many cases this oversight occurs because of 
the over reliance on a few well-worn measures especially 
knowledge testing and satisfaction surveys. Yet such an 
over reliance can easily set the stage for the presence of 
untested or even unstated assumptions regarding the true 
benefit and role of WBLI in a given contextual setting. 



evaluation in a more centralized position that examines 
the effects of a WBLI intervention within a specific 
contextual human/organizational and performance setting. 
Such a view places equal emphasis on several aspects 
affecting performance in addition to formalized planned 
learning. 

The case example provided supports this emergent 
framework because, although the TTM WBLI involved 
might be considered both efficient and effective viewed 
apart from the organizational and performance context, 
the focus group results revealed that — regardless of the 
intrinsic quality of the WBLI itself — the application 
context provided a potent influence in the success or 
failure of the instruction. Simple evaluative indicators 



ORGANIZATIONAL AND PERFORMANCE ENVIRONMENT 



EVALUATION 




TECHNOLOGICAL CAPABILITY 






HUMAN FACTORS 


•Internet and web technology 






•Change readiness 


•Computer technology 






•Motivational factors 


•Instructional design technology 






•Job complexity 



WEB-BASED LEARNING & 
INSTRUCTION (WBLI) AND 
PERFORMANCE 





Figure 2. A Framework for Evaluating WBLI. 

Based on the literature reviewed and the findings of the 
case example described, Figure 2 depicts several elements 
of an emergent evaluative framework that can be used to 
make explicit the presence of, and relationships among 
the technological, human, and organizational factors 
affecting the evaluation of WBLI. 

This framework includes a balanced view of WBLI as it is 
influenced by the steadily emerging (internet and other) 
technologies as well as the human and organizational 
context in which these are applied. Rather than make any 
given piece of instruction (web-based or otherwise) the 
central focus or unit of analysis, the framework casts 



such as pre/post engagement surveys and even 
assessments of learning (e.g., embedded quizzes, tests, 
examinations) cannot be relied upon as definitive 
evidence of WBLI merit and/or worth because these 
cannot sufficiently accommodate the organizational and 
contextual job performance environment within which 
learned knowledge is applied. 

This is further substantiated by the evidence produced by 
several relatively recent empirical studies that point to the 
critical importance of taking differences in stakeholder 
perceptions of training squarely into account as part of the 
overall evaluation process (see, e.g., Brown, 1994; 
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Michalski, 1999; Michalski & Cousins, 2000). In sum, 
the evidence from such studies (along with the case 
example discussed) support the emergent framework 
proposed in further suggesting the need to take a closer 
look at organizational factors as well as stakeholder group 
differences relative to the evaluation of WBLI. 

Conclusions 

This paper has examined the evaluation of web-based 
learning and instruction using a wider scope than is 
typically applied. The literature reviewed and the case 
example provided serve to substantiate the emergent 
evaluation framework which highlights and underscores 
the importance of human/organizational context as well as 
the technological capabilities available for developing and 
deploying WBLI. Although evaluative measures can be 
either quantitative or qualitative, the evidence suggests 
that a range of both type of measures are required to get 
an accurate view of the learning and performance 
landscape in any given organizational setting. 

Finally, instead of attempting to define a generalized 
framework to evaluate all WBLI, the emergent framework 
should be viewed as a beginning, rather than an end. It is 
certainly subject to improvement based on further 
evidence and learning. As the profound changes 
associated with advancing communications technology 
become increasingly apparent to even the casual observer, 
it is important that evaluation approaches, processes, and 
techniques continue to also evolve. Empirical work and 
shared learning based on such work holds the key to the 
future in this developing area. The continued 
investigation of WBLI evaluation, particularly within a 
range of organizational and contextual settings, is truly an 
area ripe for continued exploration. 
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